WHAT GETS TESTED GETS TAUGHT: & 
CAUTIONS FOR USING COLLEGE ADMISSIONS ws Achieve 
TESTS IN STATE ACCOUNTABILITY SYSTEMS 


Introduction 


Assessments can carry significant consequences for students (e.g., graduation exams), educators (e.g., teacher 
evaluations), and schools (e.g., school accountability), and whether one supports large-scale student testing or not, 
the research is clear that high stakes assessments impact the content educators cover and the instructional materials 
they use in their classrooms. Research on standards-based initiatives found that teachers tend to focus more on 

tested material and spend less time on the untested standards." In a 2012 survey of teachers, almost two thirds of 
those surveyed said that they skipped important topics to cover curriculum covered by assessments.” More recently, 

in a study of teachers in states that adopted both college- and career-ready standards and one of the consortia 
assessments designed to measure those standards, researchers found that 82 percent of mathematics teachers and 72 
percent of English teachers changed more than half of their instructional materials in response to changes in their state 
standards, and “85 percent of teachers increased the number of writing assignments covering specific standards and 
the content included on the consortia assessments.”? 


In an environment where assessments influence what is taught in the classroom, it is important that states and districts 
select high-quality, well-aligned assessments. States and districts must often make choices about assessments based 
on resources, time constraints, and other outside pressures. There has been a significant movement in the past several 
years towards states opting for college admissions tests, specifically the ACT or SAT, in place of a traditional summative 
assessment developed by the state. Fourteen states will administer either the ACT or SAT in 2017-18 as the statewide 
summative assessment for high school. Even more states are currently considering a change. While many states have 
historically administered the ACT or SAT to all students and included these assessments as part of a college readiness 
indicator (and many continue to do so), the adoption of a college admissions test as the statewide summative 
assessment for accountability purposes is a new development for most states. 


Using a college admissions test as the statewide summative assessment is an attractive but risky option for some 
policymakers and parents. These assessments are used for admissions by nearly all higher education institutions, are 
shorter in length than most state-designed and consortia assessments, have brand name recognition, and are known 
for predicting first-year college performance. However, notwithstanding their appeal and instrumental value for college 
admissions, neither the ACT nor College Board, the developer of the SAT, developed these tests as measures of how 
well students are meeting state mathematics and English language arts (ELA) standards, which is the primary purpose 
of state accountability tests. When they are used as a state’s mathematics and ELA tests — when they “count” for 
schools, educators, and students — there is the greatest likelihood that they will drive classroom instruction more than 
state standards do. 


This brief looks across the current evidence available on the two primary college admissions tests in order to provide 
state leaders and policymakers with the information they must consider in selecting high school assessments. The 
primary issue at hand is the alignment of college admissions tests, the ACT and SAT, to states’ college- and career-ready 
standards. 


1Stetcher et al. (2008). Pain and Gain: Implementing No Child Left Behind in Three States, 2004-2006. https://www.rand.org/con- 
tent/dam/rand/pubs/monographs/2008/RAND_MG784.pdf 

* Farkas Duffett Research Group (2012). Learning Less: Public School Teachers Describe a Narrowing Curriculum. 
http://greatminds.net/maps/documents/reports/cc-learning-less-mar12.pdf 

3 Kane et al. (2016). Teaching Higher: Educators’ Perspectives on Common Core Implementation. 
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The move toward college admissions tests in high school accountability 


Beginning nearly two decades ago, states began to administer college admissions tests to all students as part of a 
strategy to increase college access and to increase the number of students applying to and entering college, as well 

as for inclusion as a college readiness indicator.* Colorado and Illinois began administering the ACT in 2001. Five years 
later, in 2006, Maine began administering the SAT, and at the same time, the state decided to replace the statewide 
high school assessment with the SAT. After the U.S. Department of Education notified Maine that it had not provided 
sufficient documentation of the SAT’s alignment to state standards°, Maine augmented the SAT math with additional 
items®. States continued to administer the ACT and SAT to students as part of college access strategies. The passage of 
the Every Student Succeeds Act (ESSA) in 2015 provided additional assessment flexibility to states and increased the 
push to administer college admissions tests as high school accountability tests. The law allows states a new option to 
permit districts to administer “nationally recognized, locally selected” assessments, such as the SAT or ACT, in lieu of the 
state high school assessment. However, whether the test is administered by the district as a nationally recognized, locally 
selected high school assessment or by the state as the statewide high school test used for accountability, states must 
demonstrate to the U.S. Department of Education that these assessments are technically sound, valid, and reliable for 
the purposes for which they are administered. One important component includes a requirement that the assessment 
be aligned to the full range of the state’s academic content standards. 


A growing number of states have decided to use college admissions tests as the state accountability tests in ELA and 
mathematics as a way to reduce the number of tests high school students are required to take and increase student 
engagement. As of December 2017, four states (Alabama, Montana, Nebraska, and Wisconsin) have adopted the ACT 
as their statewide summative high school assessment, nine states (Colorado, Connecticut, Delaware, Illinois, Maine, 
Michigan, New Hampshire, Rhode Island and West Virginia) have adopted the SAT, and one state (Oklahoma) is allowing 
individual school districts to choose whether to administer the ACT or SAT. 


What do college admissions test measure? 


The ACT and SAT have both been recognized as predictors of college and career readiness for decades. ACT, which has 
developed its own ACT College and Career Readiness Standards, often points to its own research to claim its assessments 
are aligned to “longstanding and empirically grounded” college and career readiness standards that also align to states’ 
college- and career-ready standards. The College Board, in redesigning the SAT, examined “the major shifts taking place 
in high school instruction, standards, and assessment,” though the assessment is not aligned to any one set of state 
standards. Achieve, ACT, and the College Board have a long history working on college- and career-ready standards. 
These three organizations formed the writing team for the development of the college and career readiness Anchor 
Standards, which preceded the development of the Common Core State Standards, and both ACT and the College Board 
endorsed those standards. Today, both ACT and the College Board claim that their tests align to states’ college- and 
career-ready standards, including the Common Core State Standards. 


Three recent independent studies, using different but complementary approaches to examine alignment and other 
important issues, raise several challenges to using these college admissions tests to assess student achievement of state 
standards. Taken together, these studies point to significant challenges for states in using college admissions tests to 
measure student learning against state content standards or as a significant factor in state accountability systems. Below, 
we briefly summarize the major findings of each study, and follow that with a set of implications and recommendations 
for states. 


* Achieve (2015). Closing the Expectations Gap 2014. https://www.achieve.org/publications/closing-expectations-gap-2014 


> US Department of Education (2006). Maine Assessment Letter. https://www2.ed.gov/admins/lead/account/nclbfinalassess/me3. 


html 


6 US Department of Education (2008). Maine Assessment Letter. https://www2.ed.gov/admins/lead/account/nclbfinalassess/me6. 
html 

7 College Board. (2016). Test Specifications for the Redesigned SAT. 
https://collegereadiness.collegeboard.org/pdf/test-specifications-redesigned-sat-1.pdf Z 
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Alignment to state standards 
Alignment Study of ACT to Common Core State Standards (Achieve, 2018) 


In 2017, Achieve conducted an independent alignment study of the ACT with the Common Core State Standards. Achieve 
used an evaluation approach adapted from a methodology based on the Council of Chief State School Officers’ (CCSSO) 
Criteria for Procuring and Evaluating Large Scale Assessments®. Overall, Achieve’s review of the ACT found significant 
weaknesses across both ELA (comprising the Reading, Writing, and Essay tests) and mathematics for both content and 
depth. 


In ELA, fewer than 50 percent of items reviewed were judged to be aligned to the claimed Common Core State 
Standards. For example, many items that claimed to measure writing standards did not ask students to produce writing, 
as indicated in the state’s content standards. The ACT emphasized some aspects of reading text critically and language 
skills as well as real-world activities, but did not require students to demonstrate their abilities to write in different 
modes (argumentative, expository/nonfiction, and narrative), use text-based evidence in support of their writing, or 
focus on the vocabulary words most appropriate to high school. 


In mathematics, fewer than half of items on the assessment were judged to be aligned to the claimed Common Core 
mathematical content standards for high school. This finding, while low, may be less surprising given that ACT, in its 
technical documentation, indicates that 40-43 percent of mathematics items are designed to measure pre-8th grade 
mathematics content.® 


Delaware System of Student Assessment and Maine Comprehensive Assessment System: SAT Alignment to the 
Common Core State Standards (HumRRO, 2016)'° 


In 2016, Delaware and Maine, which adopted the SAT as their statewide summative high school assessment, 
commissioned an alignment study of the SAT from HumRRO. Both states adopted the Common Core State Standards. 
The study noted that the “SAT is reasonably aligned to the high school reading and writing portions of the CCSS, but less 
so for the math portions.”"" 


On the SAT Reading test, the study found that reviewers agreed with College Board’s alignment claims on 76 percent 
of items. In Writing/Language, 76 percent of items had item agreement. In mathematics, however, there was only 47 
percent alignment agreement between the College Board and reviewers.’” The study noted that there were a number 
of below-high school mathematics items and that the SAT did not adequately assess geometry or statistics, which are 
present in the state standards. 


The HumRRO study recommended that states using the SAT should supplement mathematics to cover content in under- 
addressed areas, particularly geometry and statistics, and supplement the number of high school level items given the 
amount of below-grade content covered. 


® Council of Chief State School Officers (2014). Criteria for Procuring and Evaluating Large Scale Assessments. https://www.ccsso. 
org/sites/default/files/2017-10/CCSSO%20Criteria%20for%20High%20Quality%20Assessments%2003242014.pdf 
° ACT. (2016). Technical Manual Supplement. https://www.act.org/content/dam/act/unsecured/documents/ACT-Technical-Manu- 
al-Supplement.pdf 
1° HumRRO. (2016). Delaware System of Student Assessment and Maine Comprehensive Assessment System: SAT Alignment to the 
Common Core State Standards. https://www.doe.k12.de.us/cms/lib/DE01922744/Centricity/Domain/414/SATalignment.pdf 
1 |bid., p. 45. 
? By comparison, an independent alignment study of Smarter Balanced to CCSS conducted by HumRRO (2016) found 87 percent 
of items fully aligned to their standard target(s) in ELA, and 81 percent of mathematics items. http://portal.smarterbalanced.org/ 
library/en/smarter-balanced-assessment-consortium-alignment-study-report.pdf 
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Feasibility of the Use of the ACT and SAT in Lieu of Florida Statewide Assessments (Assessment Solutions Group, 
2018)? 


Recently, the Florida Department of Education funded a set of studies to examine a range of issues related to using 

ACT or SAT instead of two Florida State Assessments (FSAs): specifically, the Algebra | and grade 10 ELA FSAs. The 

study examined the alignment of the ACT and SAT to the Algebra | and grade 10 ELA Florida content standards, the 
comparability between the ACT and SAT and FSAs, accommodations, accountability, and whether these assessments are 
likely to meet federal peer review criteria, which is required for using the assessments for accountability. 


The study found that neither the ACT nor SAT was fully aligned to Florida Algebra 1 standards and would require at least 
some augmentation to be aligned. For ELA, the study found that the ACT would need major adjustments. The study 
found that the test would need 10 or more items revised or replaced to be considered fully aligned with the Florida 
grade 10 content standards. The SAT reading test would need five to seven additional items to be fully aligned. 


Given Florida’s interest in potentially allowing districts to administer the ACT or SAT in lieu of the Algebra | and English 10 
FSAs, the issue of comparability across the three tests and the need to send consistent signals to educators, parents, and 
students is very important. However, the authors found that “many students would be placed at different performance 
levels on the three tests, some by as much as four out of the five performance levels...cast[ing] serious doubt on the 
interchangeability of the three tests, and the soundness of making accountability decisions based on them.’™ A set 

of additional analyses to model the use of multiple assessments within Florida’s school accountability system found 

that it would not be “fair to compare schools that use the state tests in their accountability system to those that use 

the alternate tests.”* Finally, the authors conducted a “mock federal peer review” using the information and evidence 
gathered from the multiple studies and concluded that neither the ACT nor SAT would likely meet federal assessment 
peer review requirements. 


Implications: “What Gets Tested Gets Taught” 


ACT and the College Board both claim that their assessments are aligned to states’ college- and career-ready standards, 
including states that have adopted the Common Core State Standards.** Until recently, however, there has been limited 
independent evidence available to the public to address the extent to which these assessments are aligned to states’ 
content standards in ELA and mathematics. As shown in this brief, Achieve, HumRRO, and Assessment Solutions Group 
all reported significant problems with alignment when they asked independent experts to evaluate the claims made by 
ACT and the College Board. 


These alignment issues matter, particularly when assessments are measuring student understanding of topics that are 
below grade level, as was seen in the studies described above. The danger in using admissions tests as accountability 
tests for high school is that many high school teachers will be driven to devote scarce course time to middle school 
topics, water down the high school content they are supposed to teach in mathematics, or too narrowly focus on a 
limited range of skills in ELA. To be clear, it is neither that the SAT and ACT measure unimportant topics and skills in 
mathematics and ELA, nor is it that some high school students would not benefit from additional instruction in topics 
that they should have mastered in earlier grades or focused instruction on specific skills; but students who need that 
help should get it in addition to, not instead of, the content in required high school courses. 


13 Assessment Solutions Group (2018). Feasibility of the Use of ACT and SAT in Lieu of Florida Statewide Assessments. 
http://www.trbas.com/media/media/acrobat/2018-01/70109708365300-05065523.pdf 

4 |bid., p. 6. 

45 |bid., p. 7. 

16 See, for example, How ACT Assessments Align with State College and Career Readiness Standards (2015) and College Board Suite of 
Assessments and Their Alignment to Connecticut Standards (2015) A 
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Recommendations 


States should not use the ACT or SAT as the statewide accountability measures for ELA and mathematics. States are 
using college admissions tests in school accountability in multiple ways. Some states are simply using the test as part of 
a college readiness indicator, where it is often one of multiple measures that determine student readiness for college, or 
as described earlier, as an equity measure to increase access to college for all students. These remain appropriate uses 
for college admissions tests in accountability. However, using the ACT or SAT as the primary measure of mathematics and 
ELA achievement for accountability is ill-advised for both alignment considerations and technical reasons. 


States should not allow districts to administer the ACT or SAT in lieu of its statewide summative assessment. While 
ESSA gives states the opportunity to provide districts a choice of administering a college admissions test instead of 

the statewide summative assessment, states should be cautious of opening this door. Beyond the alignment issues 
described above, there are likely to be significant comparability issues between state tests and college admissions tests. 
Additionally, with this choice, districts might “shop around” for the assessment that shows it in the best light, while 
avoiding tough but necessary conversations about the performance of all students. 


States that have adopted the ACT or SAT should ask those companies to augment their tests to improve alignment 

to send better signals to educators about instruction, and ACT and College Board should respond affirmatively. All 

the reviewed studies recommended that the college admissions tests studied be augmented with additional items to 
bring them into better alignment with state standards. They also acknowledged the challenges with augmentation, with 
Assessment Solutions Group noting that it is “a rather expensive process and adds complexity to the administration 

of the tests, since items used to augment a test need to be administered separately from the college entrance test.”?” 
However, we urge ACT and the College Board to respond positively to these recommendations, both because it will send 
more consistent signals to educators and because it will increase the likelihood of the ACT and SAT meeting peer review 
guidelines. The College Board has experience with augmenting the SAT through its experience with Maine between 2006 
and 2013. 


States that do not use the ACT or SAT as their statewide summative assessment should make sure that their current 
tests send meaningful signals about college and career readiness. States should ensure their assessments are well 
aligned to their college- and career-ready standards, and they should work with their higher education systems to use 
them to determine if students can be placed into credit-bearing courses without the need for remediation. Historically, 
one of the challenges states have faced with their high school assessments has been the disconnect between the 
assessments’ use in high schools and acceptance by institutes of higher education. The two federally-funded assessment 
consortia, PARCC and Smarter Balanced, were developed in part to provide signals to both the K-12 system and higher 
education about students’ college readiness. Several states, including New Jersey, Oregon, and Washington, allowed the 
use of PARCC or Smarter Balanced to place students into non-remedial coursework. While the number of states using 
consortia assessments has declined, there are opportunities for states using state-specific high school assessments to 
partner with their higher education institutions to use those assessments for placement decisions. California developed 
the Early Assessment Program to embed California State University placement standards into the existing California high 
school assessments to provide students with signals about their college readiness and exempt qualifying students from 
needing to take CSU placement tests, the ACT, or the SAT.*® 


1 |bid., p. 17. 
18 California State University Early Assessment Program. https://www.calstate.edu/eap/about.shtml 
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Conclusion 


College admissions tests have a long track record of providing value for institutions of higher education in providing 
predictive value for student success in entry-level college courses. However, college admissions tests were not designed 
to measure the full range of mathematics and ELA content that is reflected in state content standards. As shown 

in independent studies, these assessments have significant issues with alignment to state high school standards, 
particularly in mathematics. The misalignment between these tests and standards send the wrong signals to high school 
educators about what should be taught in mathematics and English language arts courses. States, for important equity 
reasons, should continue to find ways to offer students the opportunity to take a college admissions test, but should 
avoid the use of these assessments for accountability. 
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